Optimized anatomical structure of interest labelling

ABSTRACT

The present application describes a system ( 100 ) and method for detecting and labeling structures of interest. The system includes a current patient study database ( 102 ) containing a current patient study ( 200 ) with clinical contextual information ( 706 ). The system also includes an image metadata processing engine ( 118 ) configured to extract metadata for preparing an input for an anatomical structure classifier ( 608 ), a natural language processing engine ( 120 ) configured to extract clinical context information ( 706 ) from the prior patient documents, an anatomical structure detection and labeling engine ( 718 ), and a display device ( 108 ) configured to display findings from the current patient study. The anatomical structure detection and labeling engine ( 718 ) is configured to identify and label one or more structures of interest ( 716 ) from the extracted metadata and clinical context information ( 706 ). The processor ( 112 ) is also configured to aggregate series level data. The method detects, label and prioritize anatomical structures ( 710 ). Specifically, once patient information is received from the current patient study ( 108 ), the labeled anatomical structures ( 710 ) and the high risk anatomical structures ( 714 ) are combined to form an optimized prioritized list of structures of interest ( 716 ).

The present application relates generally to detecting and visualizingpertinent patient information and finding-specific suggestions inradiology workflow. It finds particular application in conjunction withproviding finding-specific suggestions to a radiologist of relevantanatomical structures to review in a patient based upon informationextracted from non-image data such as prior patient reports and DICOMinformation and will be described with particular reference there. Italso finds particular application in conjunction with providing thesefinding-specific suggestions to a radiologist based upon priority for aradiologist to review and will be described with particular referencethereto. However, it is to be understood that it also finds applicationin other usage scenarios and is not necessarily limited to theaforementioned application.

It has been recognized that quantitative imaging helps detect diseasesin an early stage, improve diagnosis accuracy and consistency, suggestadvanced treatment plan and guidance, and enable efficient patientfollow up. However, a very low percentage of studies were actuallyprocessed and diagnosed using advanced visualization and quantitativeimaging systems. Efforts have been made to contribute to the developmentof image visualization and processing tools. However it is quitechallenging, and often cumbersome, for clinicians to take full advantageof the imaging systems, without comprehensive training and consistentsupport. Detecting existing organs, or key anatomical structures from apatient image is quite challenging without prior knowledge of thepatient and prioritized structures to be diagnosed. On one hand, asegmentation technique may be object dependent while on the other hand,it is unknown which structures are expected, thus a global optimizationis used and the process is time consuming.

In a typical radiology interpretation work flow, given the reason forthe scan and other prior knowledge of the patient, the radiologistusually needs to identify and annotate a relevant finding. Theradiologist annotates a finding and then scans through the rest of theimages to look for other findings or related findings. The task could bequite stressful due to the limited time for an individual image and highvolume of the patients to be investigated by a physician. Currentsystems do not guide the radiologist to review other anatomicalstructures in a patient based upon a priority. This can lead to missedfindings and/or make it time consuming to detect the findings.

Additionally, due to improvements in medical imaging, the size of imagedata has significantly increased over the years (e.g. as a result of ahigher image resolution/the use of multi-temporal or multimodal data).Hence, the data retrieval process (from the image storage e.g. PACS tothe workstation) takes a non-negligible time in the workflow of theradiologist waiting to inspect the data. This is even more prevalentwhen a hospital or other medical facility uses cloud-based serviceswhere data has to be transferred from a remote server.

During review, a radiologist not only reviews the anatomical structuresin question but also wants to review related anatomical structures. Thismeans that for diagnosing a certain disease, the radiologist has a focuson specific anatomical regions, and expects the data displayed in asuitable manner (e.g. with respect to the field of view and imageorientation). Although the data loading process cannot be accelerated,the efficiency of the workflow itself can be improved, by transferringand displaying the most important data first. Consider a situation wherea radiologist has to inspect a CT scan of a cardiac patient while thelast available data set in the PACS system is a chest/abdominal scan.The present application seeks to improve the transfer of this imagedata. The system reviews the images and optimizes the data transferbased on the clinical needs. Once the images of the most relevantanatomical structures are transmitted, the remaining data would betransferred in a streaming like fashion.

The present application provides a system and method which retrieves apatient's medical data record and, using information extracted fromthese reports combined with the extracted DICOM data, provides aradiologist with the most likely affected, or high-risk, anatomicalstructures of interest (SOI). These SOIs are first segmented and thengiven to the radiologist for review. The system also uses theinformation extracted from the patient reports and the DICOM tags togenerate a probability model. The probability model presents theradiologist with additional anatomical structures that should bereviewed based upon the current findings, the reason for examination,and past historical data from other patients. These additionalanatomical structures for review are areas that are most likely to alsobe affected based upon the given information. The present applicationfurther provides utilizing a workflow-driven data transmission scheme tooptimize transmission of image data to the physician. Using the contextinformation given by the current workflow, selected imageareas/anatomical regions with the highest probability of being relevantto the examination are transmitted first. The remainingsegments/anatomical regions are given lower priority and are transmittedto the radiologist last.

The present application also provides new and improved methods andsystems which overcome the above-referenced problems and others.

In accordance with one aspect, a system for detecting and segmentingstructures of interest is provided. The system includes a currentpatient study database, a statistical model patient report database, animage metadata processing engine, a natural language processing engine,an anatomical structure detection and labeling engine, a display device,and one or more processors. The one or more processors are configured toprepare a list of suggested anatomical structures from the anatomicalstructure classifier and form a prioritized list of structures ofinterest, process the prioritized list of structures of interest throughthe anatomical structure detection and labeling engine to form anoptimized structure of interest list for the current patient study,apply the optimized structure of interest list from the current study tothe volumetric image to detect and label structures of interest, andcontrol the display device to display the optimized structures ofinterest.

In accordance with another aspect, a method for optimizing detecting andlabeling structures of interest is provided. The method extractsclinical contextual information and DICOM metadata from a currentpatient study and at least one prior patient documents, performsstatistical analysis on the extracted clinical contextual information,and employ anatomical structure classifier based on the DICOM data togenerate a list of suggested anatomical structures in the currentpatient study. The method also extracts anatomical structures from thecurrent patient study to create a patient high risk analysis report andthen detects and labels the anatomical structure. The processors combinethe suggested anatomical structures and the high risk anatomicalstructures to form an optimized prioritized list of structures ofinterest. The list is optimized and added to the volumetric image andthen displayed to the physician.

One advantage resides in improved determination of the most probableanatomical structures of interest using known patient medicalinformation and DICOM tags.

Another advantage resides in optimized transmission of image data.

Another advantage resides in improved clinical workflow.

Another advantage resides in improved patient care.

Still further advantages of the present invention will be appreciated tothose of ordinary skill in the art upon reading and understanding thefollowing detailed description.

The invention may take form in various components and arrangements ofcomponents, and in various steps and arrangement of steps. The drawingsare only for purposes of illustrating the preferred embodiments and arenot to be construed as limiting the invention.

FIG. 1 illustrates a block diagram showing a representative system forcollecting the patient data and communicating the results to thephysician.

FIG. 2 illustrates an example patient report for use in an optimizedanatomical structure of interest labeling pre-process for use indiagnostic workflow according to aspects of the present application.

FIG. 3 illustrates a flowchart of the report processing module accordingto aspects of the present application.

FIG. 4 illustrates a statistical summary chart of the most likelyinfected anatomical structures based upon information extracted from apatient clinical report according to aspects of the present application.

FIG. 5 illustrates a table indicating the likelihood of other anatomicalstructures being affected when it is known there is a positive findingin at least one anatomical finding of the patient according to aspectsof the present application.

FIG. 6 illustrates a flowchart of the image metadata processing engineaccording to aspects of the present application.

FIG. 7 illustrates a flowchart of SOI detection and labelingoptimization according to aspects of the present invention.

FIG. 8 illustrates a representative lookup table generates from thereport database processing is shown according to aspects of the presentapplication.

FIG. 9 illustrates an association among anatomical structures showingthe likelihood of other anatomical structures to be diagnosed in thesame study when it is known there is a positive finding in anotheranatomical structure.

FIG. 10 illustrates a data transmission scheme where the most relevantanatomical structure is transmitted first.

Anatomical regions in medical images are identifiable using a variety ofimage processing techniques, including classification based anatomydetection, registration using statistical templates and model-basedsegmentation or a combination of those techniques. One possibleembodiment is a sliding window approach. In this context, anatomydetection is a classification task. Using a feature based representationof a set of positive and negative image patches machine learning is usedto discriminate between the two classes. In the detection phase theclassified image is used in order to identify image regions with a highprobability for the target anatomy. Using this approach, a large numberof detectors might have to be applied to the image in order to estimatethe probabilities for all anatomies under consideration. Furthermore,the selection of suitable acceptance thresholds for the probabilities iscritical to balance the trade-off between false positive and falsenegative detections. To this end, supplemental information such as organprobabilities estimates from DICOM metadata or a report is used for theselection of the classified or for weighing of the outcome.

The present application is directed to a system and method forautomatically detecting and segmenting related anatomical structuresbased upon a patient's prior medical history, current medical issues,and related information from a prediction table. Additionally, DICOMtags are used to improve the relevant information being presented to atreating physician. The present application is inspired by the insightthat a patient's prior medical history combined known information fromother patients and DICOM tags can improve the likelihood that a treatingphysician will examine not only the area of immediate complaint but alsoreview related areas that may also be afflicted with the same or similarillness. For example, if the patient has a finding in the lung, thesystem determines all such other patients having a finding in the lungand presents to the radiologist other anatomical areas that are mostlikely affected.

Specifically, a radiologist reviews patient data from the system. Thepatient data is comprised of clinical context data and DICOM data. Theclinical context data comprises information such as the reason for thevisit or referral letter, prior reports, and any clinical indications orannotations, etc. With respect to the clinical context data, the reportscontain both information of the individual patient and information abouta select population. Since all the statements included in the reportwere confirmed by physicians, the extracted information is consideredreliable.

With reference to FIG. 1, a block diagram illustrating a representativesystem for optimizing clinical reports and presenting the information toa physician is shown. The system 100 suitably includes a current PatientStudy Database 102, a Statistical Computation Module 104, a PatientStudy Optimization Module 106, a user interface 108, interconnected viaa communications network 110. It is contemplated that the communicationsnetwork 110 includes one or more of the Internet, Intranet, a local areanetwork, a wide area network, a wireless network, a wired network, acellular network, a data bus, and the like. It should also beappreciated that the components of the system be located at a centrallocation or at multiple remote locations.

The components of the system 100 suitably include one or more processors112 executing computer executable instructions embodying the foregoingfunctionality, where the computer executable instructions are stored onmemories 114 associated with the processors 112. It is, however,contemplated that at least some of the foregoing functionality isimplemented in hardware without the use of processors. For example,analog circuitry can be employed. Further, the components of the system100 include communication units 116 providing the processors 112 aninterface from which to communicate over the communications network 110and provide the information to the physician over the user interface108. The Patient Study Optimization Module 106 includes an ImageMetadata Process engine 118, a Labeling and Segmentation Module 718, anatural language processing engine 120, and a visualization module 122,all further described in FIG. 7. The Patient Study Optimization modulealso contains an anatomy recognition module 124. This module 124receives the clinical context information from the current and priorpatient reports and indexes the information by anatomical structure ofinterest and location for future use and reference in a lookup table.Even more, although the foregoing components of the system 100 werediscretely described, it is to be appreciated that the components can becombined.

In one embodiment, a patient report is received from a current patientstudy database(s) (PACS, HIS, RIS, etc.) 102 which contain the patientdata reports and images and at least one prior patient document isretrieved from the statistical computation module 104. The documentreceived from the statistical computation module 104 contains clinicalcontextual information. The current patient report and the priorpatients' reports are received by the patient study optimization module106. The documents are reviewed, and labeled with areas where findingshave been observed. Based upon the diagnosed findings, the reports arealso used to generate a list of high risk anatomical structures. Ananatomical structure is labeled high risk if, based upon the informationreceived from the prior patient reports, there is a higher likelihoodthat based upon the areas with a finding, other anatomical structuresare also likely to have a finding. For example, in lung cancer studies,if it is known that there is a finding in the lung or the pleura, thenthere is also an 85% chance that there will be a finding in themediastinum and hila. The mediastinum and hila are marked as high riskareas and are reviewed by the radiologist first to determine adiagnosis. To fully determine this association, as described above, thepatient study optimization module 106 generates tables as laterdescribed in FIG. 4 and an association will be generated as described inFIG. 5.

With respect to FIG. 2, a patient clinical report is shown 200, forexample, a radiology report. The report contains a FINDINGS section 202that includes various body parts 204, 206, 208 and their associatedanatomical regions 210, 212, 214. For each anatomical region 210, 212,214, there is an associated statement indicating whether there has beena finding and indicates the measurement of the finding, if available.The clinical information section 216 of this report includes the reasonfor the study and related patient history.

With further reference to FIG. 3 and FIG. 4, a process flowchartdiagramming optimized SOI detection and labeling using patient clinicalreports and DICOM tags is shown. Over a period of time, an institutionmay have accumulated several patient reports like the one described inFIG. 2. A patient clinical document 100 from the patient report database300 is sent to the natural language processing engine 302 forinterpretation and extraction. The natural language processing (NLP)engine 302 extracts the clinical context information 304 and associatedbody parts listed in the patient clinical document 200. This informationis used to create a database that summarizes the extracted informationfrom the patient clinical reports including the SOI 306, the reason orthe study and patient history 308, and any findings such as measurementor modality 310. Based on this information, a module is designed todetermine and compute a statistic model 312 of the anatomical structure.The statistical modeling information is then associated with theclinical context information such as the reason for study, history andfindings 314.

FIG. 4 shows an example statistical summary report 400 of the mostlikely infected anatomical structures for a lung cancer patient. Ifpatient history information is also available, a similar table iscreated to show the most likely infected anatomical structures for thepatient with a history of a specific disease and current symptoms orfindings. If it is known that one anatomical structure is diagnosed,there is a higher probability that other anatomical structures will showsymptoms and exhibit findings.

With respect to FIG. 5 shows a probability chart 500 of other anatomicalstructures within a patient to be diagnosed when it is known that thereis a positive finding in a first anatomical structure. For lung cancerstudies, if there is a finding in the lungs and pleura, it is morelikely that the mediastinum and hila also have findings that thephysician knows to now look for.

With reference to FIG. 6, a flowchart illustrating the image metadataprocessing 600 module is shown. DICOM tags contain information relevantto anatomical structures existent in the current study such as: studydescription, protocol name, body part examined, series description,modality, contrast/bolus agent, etc. Some of these tags are studyrelated tags while others are more specific to the series or volumetricimage within the study. In order to obtain study related information,series level DICOM data is first aggregated together and then processed602. After data segregation, Bag of Words (BoW) features are constructedout of all relevant free text tags. BoW is one approach to textprocessing. The final features consist of all selected tags and the BoWfeatures. Using these BoW features, associated with statistical modelinginformation 606 identified by experts based on DICOM metadata and theirexperience, a machine learning engine 604 is configured to train a DICOMmetadata based classified/predictor 608. This module receives DICOMmetadata to represent various patient populations. The outcome is usedas an initial anatomical structure detection.

With further reference to FIG. 7, a flowchart displaying therepresentative method for retrieving clinical context information andDICOM information for a patient study is shown. For a new study, thepatient data 700 is first separated into DICOM data 702 includingmetadata 602, volumetric image data 704, and other clinical context data706. The clinical context data 706 is passed to a natural languageprocessing engine 202 to extract key statements including modality 310,reason for the study, clinical history and prior recommendations 308,SOIs 306 etc. The information is then combined with the statisticalmodel derived from the report database 312 to form the SOI list 712. Theremaining DICOM data 702 is divided into the volumetric image data 704and the DICOM metadata 602. The DICOM tags are processed through animage metadata process engine 708 where the outcome of the anatomicalstructure classifier/predictor 608 described in FIG. 6 creates theanatomical structure list 710. A high risk analysis of the anatomicalstructures is conducted for the current patient based on the patienthistorical report analysis 714. A high risk analysis for the currentpatient report will return to the radiologist the SOIs that have thehighest probability of also being affected or diagnosed with a finding.These areas are shown to the radiologist first for review. An anatomicalstructure is deemed high risk based upon the information received fromprior patient reports. This information is analyzed and combined to forma statistical analysis look up table as shown in FIG. 9. The high riskanalysis data is combined with the patient report output SOI list 712,and the anatomical structure list 710, provides a list of optimized andprioritized SOIs 716. The anatomical detection and labeling engine 718is configured and the information is combined with the optimized andprioritized SOIs 616 resulting in an optimized SOI detection andlabeling engine 720 for the current study. The optimized list is thenapplied to the volumetric labeled image data 722 to actually detect andlabel the SOIs from the current image. In one embodiment, the labelingengine 720 uses an automatic or semi-automatic segmentation routine tosegment one or more SOIs. The segmented structures can be outlined witha line around the edge, colorized, and the like. The combination of 704,720, 722 forms a visualization engine 724 which selects one or moreimage planes through the volume image to be displayed. In anotherembodiment, the images are transferred to the diagnostician based on apriority which allows the diagnostician to start reviewing the preferredimage view first while the remaining data/images are transferred.Additionally, the labeling engine 720 can select one or moresubstructures within an image (such as a heart) to transfer to thediagnostician. The substructure is transferred first and then remainingnearby substructures and complete structures are transferred later.

With reference to FIG. 8, a representative lookup table 800 generatedfrom report database processing is shown. When a physician observes afinding on an image, the finding is typically measured and recorded inthe corresponding report. The measurements are then associated with thebody part shown in the look up table. As an institution accrues more ofthese reports, a natural language processing module extracts all thefindings in the report and associates them with a body part to create adatabase. If multiple anatomical structures contain positive findingsfrom one report, then this indicates an association between theanatomical structures which is seen in the lookup table. Theassociations determined in the lookup table allow a physician to moreaccurately diagnose and treat a patient in the future. The indicatedassociations between anatomical structures in prior patient reports isused to show the likelihood of other anatomical structures likely to bediagnosed in the same study when it is known there is a positivefinding. With further reference to FIG. 9, with the inputs from thelookup table, the system calculates the statistical probability of aparticular anatomical region being diagnosed when a finding is noted900. For example, if a patient has a finding in the kidney, the systemalso determines other patients who had findings in their kidneys andretrieves findings found in other body parts.

With reference to FIG. 10, a system for retrieving and prioritizingpatient clinical images based upon current and past patient history isillustrated. During a review of a patient file, a clinician will reviewall current and past images and notations to reach a diagnosis. Theremay be many patient images to review and the images may be large files.In an effort to streamline review, a clinician can review the higherpriority images first taking into consideration the probability thatcertain anatomical structures will more likely be affected and images ofthe higher priority anatomical structures are transmitted to theclinician first. The clinician selects an image on from the patientreport 1010 on the workstation 1020 to perform a prioritization 1030.The prioritization could be performed based on data from the radiologyinformation system (RIS), hospital information system (HIS), previoususer interaction, or based upon the lookup table described in FIG. 9.The system indexes the data in the database 1040 and a communicationsnetwork 110 between the workstation 1020 and the patient clinicaldatabase 1040 allows relevant image data from the patient clinicaldatabase to be transferred to the clinician and displayed on theworkstation 1020. In the alternative, regions of interest around aspecific anatomical structure can be transferred based upon the abovedescribed prioritization.

As used herein, a processor includes one or more of a microprocessor, amicrocontroller, a graphic processing unit (GPU), anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), personal data assistant (PDA), cellular smartphones,mobile watches, computing glass, and similar body worn, implanted orcarried mobile gear. It is further contemplated that as used herein, anengine can be formed using one or more processors configured to performthe task. As further used herein, a user input device includes one ormore of a mouse, a keyboard, a touch screen display, one or morebuttons, one or more switches, one or more toggles, and the like; and adisplay device includes one or more of a LCD display, an LED display, aplasma display, a projection display, a touch screen display, and thelike.

The invention has been described with reference to the preferredembodiments. Modifications and alterations may occur to others uponreading and understanding the preceding detailed description. It isintended that the invention be constructed as including all suchmodifications and alterations insofar as they come within the scope ofthe appended claims or the equivalents thereof.

1. A system for detecting and labeling structures of interest, thesystem comprising: a patient study database including a current patientstudy database configured to store a current patient study containingclinical contextual information and a statistical model patient reportdatabase configured to store at least one or more prior patientdocuments each document including clinical contextual information; anatural language processing engine (120) configured to extractinformation about anatomical regions having a finding, such as anabnormality from the prior patient documents; wherein a statisticalanalysis is performed on the extracted information to generateassociations between an anatomical region having a finding and otheranatomical regions with a high probability of having a finding given theanatomical region has a finding.
 2. (canceled)
 3. (canceled) 4.(canceled)
 5. The system according to claim 1, wherein the prior patientdocuments contain information about the current patient study and aselected population of patients.
 6. The system according to claim 5,wherein the prior patient documents contain a findings section listingbody parts and associated anatomical regions wherein for each anatomicalregion there is an associated statement indicating whether it has afinding.
 7. (canceled)
 8. The system according to claim 1, wherein thenatural language processing engine, further processes a prior patienthistory from the current patient study to extract information aboutanatomical regions of the current patient having a finding, wherein theassociations are generated for the anatomical regions of the currentpatient having a finding.
 9. (canceled)
 10. (canceled)
 11. (canceled)12. (canceled)
 13. (canceled)
 14. (canceled)
 15. (canceled) 16.(canceled)
 17. (canceled)
 18. (canceled)
 19. (canceled)
 20. (canceled)21. The system according to claim 1, further comprising: an imagemetadata processing engine configured to extract metadata from thecurrent patient study for preparing an input for an anatomical structureclassifier.
 22. The system according to claim 1, wherein the imagemetadata processing engine is configured to use machine learning orstatistical modeling to configure the anatomical structure classifierand further includes: a DICOM metadata database containing at least oneDICOM metadata tag wherein the at least one DICOM metadata tag is pairedwith statistical modeling information of a prior patient diagnosis withsimilar properties to configure the anatomical structure classifier as aDICOM metadata anatomical structure classifier.
 23. The system accordingto claim 1, wherein the outcome of the anatomical structure classifiercreates a list of anatomical structures of the current patient.
 24. Thesystem according to claim 1, further adapted to generate a list ofstructures of interest of the current patient based on the generatedassociations and the list of anatomical structures of the currentpatient.
 25. The system according to claim 1, further comprising: ananatomical structure detection and labeling engine configured toidentify and label one or more the structures of interest in volumetricimage data from the current patient study.
 26. The system according toclaim 1, wherein the structures of interest are prioritized in the listaccording to their probability of having a finding.
 27. The systemaccording to claim 1, further comprising: a display device configured todisplay the structures of interest, wherein structures of interesthaving a higher priority are display before structures of interesthaving a lower priority.
 28. A method for optimizing detecting alabeling structures of interest, the method comprising: providing apatient study databased including a current patient study databaseconfigured to store a current patient study containing clinicalcontextual information and a statistical model patient report databaseconfigured to store at least one or more prior patient documents eachdocument including clinical contextual information; performing astatistical analysis on the extracted information to generate anassociation between an anatomical region having a finding and otheranatomical regions with a high probability of having a finding given theanatomical region has a finding.